Normalize results in results folder #18
Conversation
|
Nice! It's hard viewing the diff, we should just be careful not to delete results that are unique & if they are duplicate that it's not just the filename & model that's duplicate but also the scores (maybe checking a few files to make sure scores are within changes that may be due to randomness) |
|
@Muennighoff did a manual review of quite a few files. Naturally, as you say, it is hard to know exactly. However, almost all of the file changes are caused by moving models to a new folder, and the cases where it wasn't were primarily due to #15, where I manually resolved the diff (since you ran it with a slightly lower version, it could not be manually resolved). I have "validated" the files for MMTEB as well to ensure that they contain all the relevant splits for all the relevant tasks. Do note that this does not change any of the results in a folder without the "model_meta.json". |
…d file size of flores
…ngs-benchmark/results into normalize_results_folder
Added script to normalize results in the folder to follow the format created by
mteb run. This also merged the results if multiple folders are present (e.g. due to. #15).The path is created based on the model_meta.json so any folder without this will stay the same.
Edit: ran an additional model on a task where the results were missing
Edit: added test to ensure folder structure going forward